Reduce archived conversation disk usage with cold storage - #4016
Reduce archived conversation disk usage with cold storage#4016Quicksaver wants to merge 101 commits into
Conversation
- Preserve binary SQL values across archive round trips - Keep cold bundles authoritative until attachments restore safely - Bound restore memory and tolerate compatible schema changes
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
vouch:unvouched
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review Diff is too large for automated approval analysis. A human reviewer should evaluate this PR. You can customize Macroscope's approvability policy. Learn more. |
- Serialize archive-tree lifecycle and recheck archived shells - Restore cleanup-pending bundles before unarchive commits - Preserve retry state for writer and filesystem failures
There was a problem hiding this comment.
One Effect service convention issue found in the new ThreadColdStorage service. See the inline comment.
Posted via Macroscope — Effect Service Conventions
- Reference-count archive-tree lock users and waiters - Remove lock entries after the final operation releases them
- Define the service members inline with Context.Service - Use the inferred Service type in the layer and orchestration test
- Match archived attachments by exact persisted ids - Resume cleanup pending manifests without shell rows - Preserve attachment metadata until durable delete cleanup succeeds
- Reuse archive filtering for project rows and navigation - Cover persisted and optimistic archive visibility
- Treat persisted shell data as fast-paint cache only - Resume WebSocket events from freshly loaded snapshots
- Describe cold archive, restoration, and deletion behavior - Record sidebar consistency requirements and development ports
- Document authoritative shell refresh, event replay, and deferred cache writes - Preserve mobile archive timestamp and shell subscription safeguards
- Describe event observation as scheduling lifecycle work - Preserve best-effort preview shutdown semantics
- Drop renderer refs for removed environments - Close archived previews before queued lifecycle work - Cover catalog removal and delayed archive regressions
…nversation-data-savings
…data-savings # Conflicts: # apps/web/src/browser/ElectronBrowserHost.tsx
- Keep command receipts hot and rediscover missed deletions - Avoid stale restore replays and retain live preview baselines
- Track live restore ownership to protect active unarchives - Requeue abandoned restored manifests for cold storage - Cover restart recovery and document lifecycle behavior
…data-savings # Conflicts: # apps/server/src/provider/Layers/EventNdjsonLogger.ts # apps/web/src/components/settings/SettingsPanels.tsx # packages/client-runtime/src/rpc/client.ts
There was a problem hiding this comment.
Effect service conventions: two error-modeling findings in the new ThreadColdStorage implementation. The service tag/interface, layer wiring, and dependency acquisition otherwise look consistent with the conventions.
Posted via Macroscope — Effect Service Conventions
- Run archive cleanup under the tree lock - Structure archive validation failures
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a7332c6. Configure here.
| if (refreshedReadModel !== null) { | ||
| commandReadModel = refreshedReadModel; | ||
| } | ||
| } |
There was a problem hiding this comment.
Stale read model after partial rollback
Medium Severity
When unarchive restore succeeds but the domain command fails, rollbackRestoreTree can still commit the destructive archive boundary (hot sessions, turns, and proposed plans deleted, manifest set to cleanup_pending) and then fail in completeArchiveCleanup. That failure makes rolledBack false, so the engine skips the getCommandReadModel refresh. Event reconcile also no-ops when no unarchive event was persisted, leaving commandReadModel with phantom plan, session, and latest-turn state for later commands in the same process.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit a7332c6. Configure here.
| { | ||
| discard: true, | ||
| }, | ||
| ); |
There was a problem hiding this comment.
Rollback archive skips quiescing
Medium Severity
rollbackRestoreTree re-archives through archiveImpl with an empty quiesce effect. After a failed unarchive of a still-hot archived shell, provider sessions, terminals, and log writers may still be active because the lifecycle archive job never ran, so rollback can delete hot rows and history while writers continue.
Reviewed by Cursor Bugbot for commit a7332c6. Configure here.
|
Closing in favor of #2829 (orchestration V2). #2829 deletes the V1 orchestration layer this PR builds on — This is not a judgement on the change itself. Several of these are real gaps we still want fixed; the base just moved out from under them. Once #2829 merges, please rebase onto |
2 participants
Summary
This adds compressed cold storage for archived conversations. Once a thread is archived, its conversation data and attachments are moved out of the hot state database into a dedicated
archive.sqlitebundle, while unarchiving restores the complete thread tree before applying the domain command.The lifecycle is durable and restart-safe: migrations 035/036 queue existing archived and deleted threads, background work rediscovers missed archive/delete jobs, and restoration keeps the cold bundle authoritative until the unarchive command commits successfully. Command receipts stay hot for retry idempotency until permanent deletion, while attachment ownership is preserved from exact persisted IDs and cold-bundle filenames throughout archive, restore, and durable deletion.
Archived thread shells remain consistent across clients without relying on compacted event history: each WebSocket session reconciles an authoritative shell snapshot, then replays live changes without losing archive removals during the handoff. Archive and deletion lifecycle events also close server preview sessions and clear renderer preview state so unarchive starts without stale mini-players, capture sessions, or desktop tabs.
What Changed
archive.sqlite.restoredmanifests as reservations instead of replaying stale bundle rows and attachments over an already-active conversation.restoredreservations abandoned across process restarts while preserving process-local ownership for an unarchive that is still in flight.cleanup_pendingbundles, and prevented queued archive work or post-commit failures from undoing a successful restore.cleanup_pendingmanifests even when the archived shell row is missing, while preserving exact attachment metadata until durable delete cleanup succeeds.t3-sqlite-stateinspection path for both hotstate.sqlitedata and coldarchive.sqlitemanifests/chunks.Why
With fairly moderate usage, the user
.t3folder grew to more than 30 GB. That is far too much local data growth for normal use, and without lifecycle controls it gets out of hand very quickly. Archived and deleted conversations need to stop accumulating indefinitely in hot storage while remaining reliably recoverable when a user unarchives them.Validation
pnpm exec vp checkpassed with exit 0; it reported 9 pre-existing lint warnings outside the changed files.pnpm exec vp run typecheckpassed with exit 0.pnpm exec vp run lint:mobilepassed with exit 0; optional SwiftLint, ktlint, and detekt checks were skipped because those tools are not installed locally.HttpResponseCompression.ts(Response.body).Proof
No attached visual proof is necessary; the data lifecycle and sidebar interaction are covered by the automated and isolated Playwright validation above.
Note
Reduce archived conversation disk usage with cold storage archival and cache eviction
archive.sqlitedatabase; theThreadDeletionReactoris extended into a general lifecycle reactor that enqueues and processes archive, delete, and compaction jobs.thread_archive_manifestsandthread_cleanup_queuetables and backfill existing archived/deleted threads into the new queues.Macroscope summarized a7332c6.